This invention relates generally to analysis of events in a system, and more particularly to automated identification and visualization of event outliers and outage detection.
Many different types of systems generate data streams that provide near-continuous event data describing events occurring on the systems. Examples include network monitoring, click streams, web server log records, packet flow data of internet providers, phone logs, sensor data streams, traffic monitoring, credit card transactions, financial tickers, video surveillance, and power usage. When problems develop with these systems, these problems can greatly impact the revenue and reputation of a company. These underlying problems may present themselves in the underlying data stream, but are challenging to identify, making timely detection difficult and delaying an appropriate investigation and response.
Outlier detection in continuous data streams can differ from outlier detection of a static data set. Typically, the large data size of a continuous data stream prohibits disk storage, the large data volume influx prohibits passing over the data more than once, the continuous nature of streaming data results in an unbounded time series, and streaming data often evolves over time. In addition, detection of outliers in the data set near the time that they occur (i.e., before they become stale) is critical to a timely response, while an alert based on a detection of an outlier should prevent false alarms. In addition, outliers may be caused by many reasons, only some of which may represent a problem with the monitored systems. Without a thorough analysis of these outliers, there is a risk of over or under reporting outliers that are not related to system outages.